Systems and methods for automated segmentation of patient specific anatomies for pathology specific measurements

ABSTRACT

Systems and methods are provided for multi-schema analysis of patient specific anatomical features from medical images. The system may receive medical images of a patient and metadata associated with the medical images indicative of a selected pathology, and automatically classify the medical images using a segmentation algorithm. The system may use an anatomical feature identification algorithm to identify one or more patient specific anatomical features within the medical images by exploring an anatomical knowledge dataset. A 3D surface mesh model may be generated representing the one or more classified patient specific anatomical features, such that information may be extracted from the 3D surface mesh model based on the selected pathology. Physiological information associated with the selected pathology for the 3D surface mesh model may be generated based on the extracted information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International PCT PatentApplication Serial No. PCT/IB2022/051216, filed Feb. 10, 2022, whichclaims the benefit of priority of GB Patent Application Serial No.2101908.8, filed Feb. 11, 2021, the entire contents of each of which areincorporated herein by reference.

FIELD OF USE

The present disclosure is directed to systems and methods formulti-schema analysis of patient specific anatomical features frommedical images for pathology specific measurements for specific usecases in diagnosis, planning and treatment.

BACKGROUND

Creating accurate 3D models of specific parts of a patient's anatomy ishelping to transform surgery procedures by providing insights toclinicians for preoperative planning. Benefits include, for example,better clinical outcomes for patients, reduced time and costs forsurgery and the ability for patients to better understand a plannedsurgery.

However, there is still a need to provide 3D models providing greaterinsight on the patient anatomy or pathology.

In view of the foregoing drawbacks of previously known systems andmethods, there exists a need for enhanced systems and methods foranalyzing medical images of a patient to create 3D models to assist indiagnosis, planning, and/or treatment.

SUMMARY

The present disclosure overcomes the drawbacks of previously-knownsystems and methods by providing systems and methods for multi-schemaanalysis of patient specific anatomical features from medical images forpathology specific measurements for specific use cases in diagnosis,planning, and/or treatment.

The generation of a scale virtual replica of the patient's anatomy,e.g., a 3D anatomical model, is an extremely useful tool that can beused to drive personalized patient specific decisions in clinicalpractice, e.g., for pre-operative planning. The present disclosuredemonstrates how to generate patient specific 3D models of a patient'scomplete anatomy, for example, by building machine learning models toautomatically detect and segment anatomy from medical scans. Thesemodels may be trained using curated semantically labeled datasets. Toproduce a 3D segmentation, a neural network or machine learningalgorithm is trained to identify the anatomical features within a set ofmedical images. These images are semantically labeled with the locationof the anatomical features and their constituent parts and landmarks.Accordingly, the segmentation algorithm can take new datasets and theircomplementary landmarks and use those to identify new anatomicalfeatures or landmarks.

The segmentation process is the first step in producing patient specificinsights into anatomical features, which power decision making in theclinical setting. The technology made available by Axial MedicalPrinting Limited, Belfast, United Kingdom, turns the 2D medical scansinto scale 3D models of the patient's anatomy, which allows 3D decisionmaking and understanding. The output of the segmentation process is theprecise set of coordinates that represent the anatomical features in thescan. This representation of the anatomy allows definitive statementsabout the features to be made, for example, standard measurements suchas size, length, volume, diameter, oblique cross-section and others. Asa result, the shape and location of the anatomical feature or pathologymay be calculated and incorporated into a personalized decision makingprocess by the surgeon. These measurements may be used to drive criticaldecisions about the patient's condition and any proposed intervention.

More significantly the systems described herein can distinguish betweennormal and pathological states of the anatomy and any anatomicalfeature. The training process may be further embellished with thisinformation and may use this to drive further classes of anatomicalfeatures. For example, blood may be identified and segmented within amedical scan. By incorporating information about the pathological state,blood clots also may be identified and segmented within a vessel suchthat the type and severity of the pathology may be identified. Incombination with measurement data about the anatomy, this information iscrucial for decision making in acute blood clot based pathologies suchas stroke or coronary disease.

Pathology specific patentable artifacts may be created by combining theauto-segmentation algorithms described herein with large labeledtraining datasets that are specific to each pathology, such that thecombination of the appropriate algorithm and the specific data createsunique sets of artifacts per pathology. The ability to provide specificgrouping of functionalities of a segmentation provides significantbenefits to specific clinical problems. Moreover, the ability to providethe automated segmentation also opens up a number of pathology specificapplications that would benefit from the systems described herein.

In accordance with one aspect, a method for multi-schema analysis ofpatient specific anatomical features from medical images is provided.The method may include: receiving, by a server, medical images of apatient and metadata associated with the medical images indicative of aselected pathology; automatically processing, by the server, the medicalimages using a segmentation algorithm to label pixels of the medicalimages and to generate scores indicative of a likelihood that the pixelswere labeled correctly; using, by the server, an anatomical featureidentification algorithm to probabilistically match associated groups ofthe labeled pixels against an anatomical knowledge dataset to classifyone or more patient specific anatomical features within the medicalimages; generating, by the server, a 3D surface mesh model defining asurface of the one or more classified patient specific anatomicalfeatures; extracting, by the server, information from the 3D surfacemesh model based on the selected pathology; and generating, by theserver, physiological information associated with the selected pathologyfor the 3D surface mesh model based on the extracted information. Forexample, the information extracted from the 3D surface mesh model mayinclude a 3D surface mesh model of an anatomical feature isolated fromthe one or more classified patient specific anatomical features based onthe selected pathology.

Generating, by the server, physiological information associated with theselected pathology for the 3D surface mesh model may include:determining start and end points of the isolated anatomical feature;taking slices at predefined intervals along an axis from the start pointto the end point; calculating a cross-sectional area of each slicedefined by a perimeter of the isolated anatomical feature; extrapolatinga 3D volume between adjacent slices based on the respectivecross-sectional areas; and calculating an overall 3D volume of theisolated anatomical feature based on the extrapolated 3D volumes betweenadjacent slices.

Generating, by the server, physiological information associated with theselected pathology for the 3D surface mesh model may include:determining start and end points of the isolated anatomical feature anda direction of travel from the start point to the end point; raycastingat predefined intervals along an axis in at least three directionsperpendicular to the direction of travel and determining distancesbetween intersections of each ray cast and the 3D surface mesh model;calculating a center point at each interval by triangulating thedistances between intersections of each ray cast and the 3D surface meshmodel; adjusting the direction of travel at each interval based on adirectional vector between adjacent calculated center points, such thatraycasting at the predefined intervals occur in at least threedirections perpendicular to the adjusted direction of travel at eachinterval; and calculating a centerline of the isolated anatomicalfeature based on the calculated center points from the start point tothe end point.

Generating, by the server, physiological information associated with theselected pathology for the 3D surface mesh model may include:calculating a centerline of the isolated anatomical feature; determiningstart and end points of the isolated anatomical feature and adirectional vector from the start point to the end point; establishingcutting planes at predefined intervals along the centerline based on thedirectional vector from the start point to the end point, each cuttingplane perpendicular to a direction of travel of the centerline at eachinterval; raycasting in the cutting plane at each interval to determinea position of intersection on the 3D surface mesh model from thecenterline; and calculating a length across the 3D surface mesh modelbased on the determined positions of intersection at each interval.

Generating, by the server, physiological information associated with theselected pathology for the 3D surface mesh model may include:determining start and end points of the isolated anatomical feature;taking slices at predefined intervals along an axis from the start pointto the end point; calculating a cross-sectional area of each slicedefined by a perimeter of the isolated anatomical feature; andgenerating a heat map of the isolated anatomical feature based on thecross-sectional area of each slice.

Generating, by the server, physiological information associated with theselected pathology for the 3D surface mesh model may include:determining start and end points of the isolated anatomical feature;calculating a centerline of the isolated anatomical feature; determininga directional travel vector between adjacent points along thecenterline; calculating a magnitude of change of directional travelvectors between adjacent points along the centerline; and generating aheat map of the isolated anatomical feature based on the magnitude ofchange of directional travel vectors between adjacent points along thecenterline.

In some embodiments, the generated physiological information associatedwith the selected pathology for the 3D surface mesh model may include anassociated timestamp, such that the method further includes: recording,by the server, the generated physiological information and theassociated timestamp; and calculating, by the server, changes betweenthe recorded physiological information over time based on associatedtimestamps, indicative of progression of the selected pathology.Accordingly, the method further may include: calculating, by the server,a magnitude of the changes between the recorded physiologicalinformation over time; and generating, by the server, a heat map of theisolated anatomical feature based on the magnitude of the changesbetween the recorded physiological information over time.

Extracting, by the server, information from the 3D surface mesh modelbased on the selected pathology may include: isolating an anatomicalfeature from the one or more classified patient specific anatomicalfeatures based on the selected pathology; analyzing features of theisolated anatomical feature with an anatomical feature database toidentify one or more landmarks of the isolated anatomical feature;associating the one or more identified landmarks with the pixels of themedical images; and generating a 3D surface mesh model defining asurface of the isolated anatomical feature comprising the identifiedlandmarks. Moreover, the method may further include: identifying, by theserver, a guided trajectory for performing a surgical procedure from asurgical implement database based on the selected pathology and the oneor more identified landmarks; and displaying the guided trajectory to auser.

In addition, the method further may include: receiving, by the server,patient demographic data; identifying, by the server, one or moremedical devices from a medical device database based on the patientdemographic data and the generated physiological information associatedwith the selected pathology for the 3D surface mesh model; anddisplaying the identified one or more medical devices to a user.Moreover, the method further may include: receiving, by the server,patient demographic data; identifying, by the server, one or moretreatment options from a surgical implement database based on thepatient demographic data and the generated physiological informationassociated with the selected pathology for the 3D surface mesh model;and displaying the identified one or more treatment options to a user.

Extracting, by the server, information from the 3D surface mesh modelbased on the selected pathology may include: isolating an anatomicalfeature from the one or more classified patient specific anatomicalfeatures based on the selected pathology; analyzing features of theisolated anatomical feature with an anatomical feature database toidentify one or more landmarks of the isolated anatomical feature;analyzing features of the one or more landmarks with a referencefracture database to detect a fracture of the isolated anatomicalfeature; and generating a 3D surface mesh model of the isolatedanatomical feature comprising the one or more identified landmarks andthe detected fracture. Accordingly, the method further may includematching the 3D surface mesh model of the isolated anatomical featureagainst the reference fracture database to classify the detectedfracture.

The method further may include: delineating, by the server, theclassified one or more patient specific anatomical features into binarylabels; separating, by the server, the binary labels into separateanatomical features; and mapping, by the server, the separate anatomicalfeatures to original grey scale values of the medical images andremoving background within the medical images, and wherein the generated3D surface mesh model defines a surface of the separate anatomicalfeatures, or comprises a volumetric render defined by mapping specificcolors or transparency values to the classified one or more patientspecific anatomical features. In some embodiments, the segmentationalgorithm may include at least one of a threshold-based, decision tree,chained decision forest, or neural network method. The physiologicalinformation associated with the selected pathology may include at leastone of diameter, volume, density, thickness, surface area, HounsfieldUnit standard deviation, or average.

In accordance with another aspect of the present disclosure, a systemfor multi-schema analysis of patient specific anatomical features frommedical images is provided. The system may include a server and may:receive medical images of a patient and metadata associated with themedical images indicative of a selected pathology; automatically processthe medical images using a segmentation algorithm to label pixels of themedical images and to generate scores indicative of a likelihood thatthe pixels were labeled correctly; use an anatomical featureidentification algorithm to probabilistically match associated groups ofthe labeled pixels against an anatomical knowledge dataset to classifyone or more patient specific anatomical features within the medicalimages; generate a 3D surface mesh model defining a surface of the oneor more classified patient specific anatomical features; extractinformation from the 3D surface mesh model based on the selectedpathology; and generate physiological information associated with theselected pathology for the 3D surface mesh model based on the extractedinformation. For example, the information extracted from the 3D surfacemesh model may include a 3D surface mesh model of an anatomical featureisolated from the one or more classified patient specific anatomicalfeatures based on the selected pathology.

In accordance with yet another aspect of the present disclosure, anon-transitory computer-readable memory medium having instructionsstored thereon is provided, that when loaded by at least one processorcause the at least one processor to: receive medical images of a patientand metadata associated with the medical images indicative of a selectedpathology; automatically process the medical images using a segmentationalgorithm to label pixels of the medical images and to generate scoresindicative of a likelihood that the pixels were labeled correctly; usean anatomical feature identification algorithm to probabilisticallymatch associated groups of the labeled pixels against an anatomicalknowledge dataset to classify one or more patient specific anatomicalfeatures within the medical images; generate a 3D surface mesh modeldefining a surface of the one or more classified patient specificanatomical features; extract information from the 3D surface mesh modelbased on the selected pathology; and generate physiological informationassociated with the selected pathology for the 3D surface mesh modelbased on the extracted information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows some example components that may be included in anmulti-schema analysis platform in accordance with the principles of thepresent disclosure.

FIG. 2 is a flow chart illustrating exemplary method steps formulti-schema analysis of patient specific anatomical features frommedical images in accordance with the principles of the presentdisclosure.

FIG. 3 is a flow chart illustrating exemplary method steps forgenerating volume measurements of a patient specific anatomical featurein accordance with the principles of the present disclosure.

FIG. 4A illustrates cross-sectional area measurements at various pointsalong a vessel, and FIG. 4B illustrates determination of volume based onthe cross-sectional area measurements in accordance with the principlesof the present disclosure.

FIG. 5 is a flow chart illustrating exemplary method steps forgenerating centerline measurements of a patient specific anatomicalfeature in accordance with the principles of the present disclosure.

FIG. 6 illustrates determination of a centerline in accordance with theprinciples of the present disclosure.

FIG. 7A illustrates center points of a vessel, FIG. 7B illustrates acenterline of the vessel, FIG. 7C illustrates measurement of length ofthe centerline of the vessel, and FIG. 7D illustrates the vesseldepicted across medical images.

FIG. 8A illustrates start and end points of a patient specificanatomical feature, FIG. 8B illustrates a centerline of the patientspecific anatomical feature, FIG. 8C illustrates the centerlines ofvarious patient specific anatomical features, and FIG. 8D illustratesthe centerlines of a network of patient specific anatomical features.

FIG. 9 is a flow chart illustrating exemplary method steps forgenerating surface length measurements of a patient specific anatomicalfeature in accordance with the principles of the present disclosure.

FIG. 10 illustrates determination of a surface length in accordance withthe principles of the present disclosure.

FIG. 11 illustrates a surface length of a patient specific anatomicalfeature.

FIG. 12 is a flow chart illustrating exemplary method steps forgenerating a heat map of a patient specific anatomical feature based onvolume in accordance with the principles of the present disclosure.

FIGS. 13A and 13B illustrate volume-based heat maps of a patientspecific anatomical feature.

FIG. 14 is a flow chart illustrating exemplary method steps forgenerating a heat map of a patient specific anatomical feature based ontortuosity in accordance with the principles of the present disclosure.

FIG. 15 illustrates a tortuosity-based heat map of a patient specificanatomical feature.

FIG. 16 is a flow chart illustrating exemplary method steps forgenerating a 3D surface mesh model of a patient specific anatomicalfeature with identified landmarks in accordance with the principles ofthe present disclosure.

FIG. 17A illustrates exemplary method steps for mapping identifiedlandmarks of a patient specific anatomical feature to a 3D surface meshmodel in accordance with the principles of the present disclosure.

FIG. 17B illustrates identified landmarks of a patient specificanatomical feature mapped to a 3D surface mesh model.

FIG. 18 is a flow chart illustrating exemplary method steps foridentifying medical devices and treatment options for a pathology inaccordance with the principles of the present disclosure.

FIG. 19A illustrates a pathology of a bone, and FIGS. 19B and 19Cillustrate various medical devices that may be used for treatment of thepathology.

FIG. 20 is a flow chart illustrating exemplary method steps fordetecting and classifying a fracture of a patient specific anatomicalfeature in accordance with the principles of the present disclosure.

FIGS. 21A to 21D illustrate mapping a detected fracture of a patientspecific anatomical feature to a 3D surface mesh model in accordancewith the principles of the present disclosure.

FIG. 22 is a flow chart illustrating exemplary method steps for trackingprogression of a pathology over time in accordance with the principlesof the present disclosure.

FIGS. 23A to 23F illustrate various progressions of pathologies overtime.

FIGS. 24A to 24F illustrate heat maps of various progressions ofpathologies over time.

FIG. 25 is a flow chart illustrating exemplary method steps foranalyzing physiological parameters of separate anatomical features inaccordance with the principles of the present disclosure.

FIG. 26 illustrates generation of 3D volumetric rendering of separateanatomical features in accordance with the principles of the presentdisclosure.

FIG. 27A illustrates an original medical image of a patient specificanatomical feature, FIG. 27B illustrates separate anatomical featuresoverlaid on the original medical image, FIG. 27C illustrates theseparate anatomical features with the background removed, and FIG. 27Dillustrates a 3D volumetric rendering of the separate anatomicalfeatures.

FIGS. 28A and 28B illustrate exemplary method steps for measuring anocclusion of a patient specific anatomical feature in accordance withthe principles of the present disclosure.

FIG. 29 is a flow chart illustrating exemplary method steps foranalyzing physiological parameters of separate anatomical features inaccordance with the principles of the present disclosure.

FIGS. 30A to 30E illustrate generating measurements of a patientspecific anatomical feature in accordance with the principles of thepresent disclosure.

FIG. 31 illustrates weight masks generated with the Euclidean distanceweight approach, as well as their effect on the loss function inaccordance with the principles of the present disclosure.

FIG. 32 illustrates various segmentations of bone within medical imagesfor training purposes in accordance with the principles of the presentdisclosure.

FIG. 33 illustrates various segmentations of a myocardium within medicalimages of ground truth data for training purposes in accordance with theprinciples of the present disclosure.

DETAILED DESCRIPTION

Referring to FIG. 1 , components that may be included in multi-schemaanalysis platform 100 are described. Platform 100 may include one ormore processors 102, communication circuitry 104, power supply 106, userinterface 108, and/or memory 110. One or more electrical componentsand/or circuits may perform some of or all the roles of the variouscomponents described herein. Although described separately, it is to beappreciated that electrical components need not be separate structuralelements. For example, platform 100 and communication circuitry 104 maybe embodied in a single chip. In addition, while platform 100 isdescribed as having memory 110, a memory chip(s) may be separatelyprovided.

Platform 100 may contain memory and/or be coupled, via one or morebuses, to read information from, or write information to, memory. Memory110 may include processor cache, including a multi-level hierarchicalcache in which different levels have different capacities and accessspeeds. The memory may also include random access memory (RAM), othervolatile storage devices, or non-volatile storage devices. Memory 110may be RAM, ROM, Flash, other volatile storage devices or non-volatilestorage devices, or other known memory, or some combination thereof, andpreferably includes storage in which data may be selectively saved. Forexample, the storage devices can include, for example, hard drives,optical discs, flash memory, and Zip drives. Programmable instructionsmay be stored on memory 110 to execute algorithms for automaticallysegmenting and identifying patient specific anatomical features withinmedical images, including corresponding anatomical landmarks, generating3D surface mesh models of the patient specific anatomical features, andextracting information from the 3D surface mesh models to generatephysiological information of the patient specific anatomical featuresbased on selected pathologies.

Platform 100 may incorporate processor 102, which may consist of one ormore processors and may be a general purpose processor, a digital signalprocessor (DSP), an application specific integrated circuit (ASIC), afield programmable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anysuitable combination thereof designed to perform the functions describedherein. Platform 100 also may be implemented as a combination ofcomputing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration.

Platform 100, in conjunction with firmware/software stored in the memorymay execute an operating system (e.g., operating system 124), such as,for example, Windows, Mac OS, Unix or Solaris 5.10. Platform 100 alsoexecutes software applications stored in the memory. For example, thesoftware may be programs in any suitable programming language known tothose skilled in the art, including, for example, C++, PHP, or Java.

Communication circuitry 104 may include circuitry that allows platform100 to communicate with an image capture device and/or other computingdevices for receiving image files, e.g., 2D medical images, and metadataassociated therewith indicative of a patient specific pathology.Additionally or alternatively, image files may be directly uploaded toplatform 100. Communication circuitry 104 may be configured for wiredand/or wireless communication over a network such as the Internet, atelephone network, a Bluetooth network, and/or a WiFi network usingtechniques known in the art. Communication circuitry 104 may be acommunication chip known in the art such as a Bluetooth chip and/or aWiFi chip. Communication circuitry 104 permits platform 100 to transferinformation, such as 3D surface mesh models, physiological measurements,and treatment options, locally and/or to a remote location such as aserver.

Power supply 106 may supply alternating current or direct current. Indirect current embodiments, power supply may include a suitable batterysuch as a replaceable battery or rechargeable battery and apparatus mayinclude circuitry for charging the rechargeable battery, and adetachable power cord. Power supply 106 may be charged by a charger viaan inductive coil within the charger and inductive coil. Alternatively,power supply 106 may be a port to allow platform 100 to be plugged intoa conventional wall socket, e.g., via a cord with an AC to DC powerconverter and/or a USB port, for powering components within platform100.

User interface 108 may be used to receive inputs from, and/or provideoutputs to, a user. For example, user interface 108 may include atouchscreen, display, switches, dials, lights, etc. Accordingly, userinterface 108 may display information such as 3D surface mesh models,physiological measurements, heat maps, list of available medical devicesfor a patient specific pathology, treatment options, etc. to facilitatediagnosis, preoperative planning, and treatment for specific use cases,as described in further detail below. Moreover, user interface 108 mayreceive user input including patient demographic data, e.g., patientsize, age, weight, medical history, patient specific pathologies, etc.,and feedback from the user based on information displayed to the user,e.g., corrected anatomical feature identification, physiologicalmeasurements, specific anatomical feature selection, such that platform100 may adjust the information accordingly. In some embodiments, userinterface 108 is not present on platform 100, but is instead provided ona remote, external computing device communicatively connected toplatform 100 via communication circuitry 104.

Memory 110, which is one example of a non-transitory computer-readablemedium, may be used to store operating system (OS) 124, image receivermodule 112, segmentation module 114, anatomical feature identificationmodule 116, 3D surface mesh model generation module 118, anatomicalfeature information extraction module 120, and physiological informationgeneration module 122. The modules are provided in the form ofcomputer-executable instructions that may be executed by processor 102for performing various operations in accordance with the disclosure.

Image receiver module 112 may be executed by processor 102 for receivingstandard medical images, e.g., 2D and/or 3D medical images, of one ormore patient specific anatomical features taken from one or acombination of the following: CT, MRI, PET, and/or SPCET scanner. Themedical images may be formatted in a standard compliant manner such aswith DICOM. The medical images may include metadata embedded thereinindicative of a patient specific pathology associated with the patientspecific anatomical features in the medical images. Image receivermodule 112 may pre-process the medical images for further processing andanalysis as described in further detail below. For example, the medicalimages may be pre-processed to generate a new set of medical imageswhich are evenly distributed according to a predetermined orientationbased on the patient specific anatomic feature, specific pathology ofthe patient, or any downstream application such as preoperative trainingand/or for machine learning/neural network training purposes. Moreover,image receiver module 112 may receive medical images takensimultaneously from multiple perspectives of a patient specificanatomical feature to enhance segmentation of the patient specificanatomical features.

Segmentation module 114 may be executed by processor 102 for automatedsegmentation of the medical images received by image receiver module112, e.g., to assign a label to each pixel of the medical images. Theassigned label may represent a specific tissue type, e.g., bone, softtissue, blood vessel, organ, etc. Specifically, segmentation module 114may use machine learning based image segmentation techniques includingone or a combination of the following techniques: threshold-based,decision tree, chained decision forest, or a neural network method, suchthat the results of each technique may be combined to produce a finalsegmentation result, as described in U.S. Pat. No. 11,138,790 and U.S.Patent Appl. Pub. No. 2021/0335041 to Haslam, both assigned to theassignee of the present disclosure, and both incorporated herein intheir entireties by reference. The machine learning based imagesegmentation techniques may be trained using a knowledge databaseincluding pre-labeled medical images (i.e., ground truth data).

For example, segmentation module 114 may apply a first segmentationtechnique, e.g., a threshold-based segmentation, to assign a label toeach pixel of the medical images based on whether a characteristic,e.g., Hounsfield value, of the pixel meets/exceeds a predeterminedthreshold. The predetermined threshold may be determined via, e.g.,histogram analysis, as described in U.S. Pat. No. 11,138,790.Segmentation module 114 further may expand on the threshold-basedsegmentation technique by using a logistic or probabilistic function tocalculate a score as to the likelihood of a pixel being the tissue typeas labeled by the threshold-based segmentation.

Segmentation module 114 may then apply a decision tree to each labeledpixel of the medical images to thereby classify/label each pixel basedat least in part on, but not solely on, the score. As described in U.S.Pat. No. 11,138,790, the decision tree may be applied to a subset of thelabeled pixels by subsampling the medical images, such that segmentationmodule 114 may recover full segmentation of the medical images by usingstandard interpolation methods to up-scale the labeled pixels of thesubset of pixels of the medical images. The decision tree may consider,for each pixel, the score as well as, for example, the followingproperties: how many pixels looking almost like bone are near the pixelin question; how many pixels looking exactly like bone are near thepixel in question; or how strong is an overall gradient of the image atthe given pixel. For example, if a pixel in question is labeled as bonewith a score of 60/100, the first decision node of the decision tree canask how many pixels looking almost like bone are near the pixel inquestion. If the answer is close to zero, meaning that very few pixelsnear the pixel in question look almost like bone, segmentation module114 may determine that the pixel in question is not bone, even thoughthe previous bone label had a score of 60/100. A new score may then begenerated as to the likelihood that the pixel in question was correctlylabeled by the decision tree algorithm. Accordingly, applying thedecision tree to the pixels of the medical images may produce moreaccurate final segmentation results with less noise. As will beunderstood by a person having ordinary skill in the art, the decisiontree may consider other properties that may be useful in determining alabel for the pixel.

Additionally or alternatively, segmentation module 114 may apply achained decision forest, in which the results of an initial/previousdecision tree and the results of another segmentation technique, e.g., aNeural Network, for the same pixel in question may be fed to a newdecision tree along with the scores associated with the results. Forexample, the new decision tree may ask one or more questions asdescribed above to determine whether each of the previous segmentationtechniques correctly labeled the pixel in question. Thus, if theinitial/previous decision tree labeled the pixel in question as bone;whereas, the Neural Network labeled the pixel in question as not bone,the new decision tree may determine that the pixel in question is bonebased on the responses to the one or more questions asked by the chaineddecision forest, such that the label allocated by the Neural Network forthe pixel in question is discarded. Moreover, each forest-node may betreated as a simple classifier that produces a score as to how likelythe pixel was correctly labeled by each subsequent new decision tree.Accordingly, applying the chained decision forest to the pixels of themedical images may produce more accurate final segmentation results.

Anatomical feature identification module 116 may be executed byprocessor 102 for identifying one or more patient specific anatomicalfeatures within the medical images by probabilistically matching thepixels labeled by segmentation module 114 against an anatomicalknowledge dataset within the knowledge database. Specifically, asdescribed in U.S. Pat. No. 11,138,790, anatomical feature identificationmodule 116 initially may group the pixels labeled by segmentation module114, e.g., by establishing links between the differentlabeled/classified pixels based on similarities between the labeledpixels. For example, all the pixels labeled “bone” may be grouped/linkedtogether in a first group, all the pixels labeled “organ” may begrouped/linked together in a second group, and all the pixels labeled“blood vessel” may be grouped/linked together a third group.

Anatomical feature identification module 116 may then use an anatomicalfeature identification algorithm to explore the anatomical knowledgedataset to identify the patient specific anatomical features within themedical images by establishing links between the grouped labeled pixelswith existing knowledge within the anatomical knowledge dataset. Forexample, the existing knowledge may include known information regardingvarious anatomic features such as tissue types, e.g., bone, bloodvessel, or organ, etc., represented as nodes within a graph database ofthe anatomical knowledge dataset, as well as pre-labeled ground truthdata that may be used to train the various segmentation algorithms.

The medical ontology of the existing knowledge of anatomic featureswithin the graph database may be represented as a series of nodes whichare grouped together through at least one of: functions, proximity,anatomical groupings, or frequency of appearance in the same medicalimage scan. For example, nodes representing an organ may be groupedtogether as a heart because they are within a predetermined proximity toeach other, are all near nodes representing a blood vessel which aregrouped together as an aorta, and have a high frequency of appearance inthe same medical image scan. Accordingly, the anatomical featureidentification algorithm may identify the patient specific anatomicalfeature within the medical image through exploration of the graphdatabase to determine which group of nodes most resemble the groupedlabeled pixels, e.g., based on the established links between the groupedlabeled pixels and the group of nodes. Anatomical feature identificationmodule 116 further may generate a score representing the likelihood thatthe patient specific anatomical feature was correctly identified by theanatomical feature identification algorithm.

3D surface mesh model generation module 118 may be executed by processor102 for generating a 3D surface mesh model of the patient specificanatomical features within the medical images based on the results ofthe segmentation algorithm as well as the results of the anatomicalfeature identification algorithm, described above, and for extracting a3D surface mesh model from the scalar volumes to generate a 3D printablemodel. For example, as described in U.S. Pat. No. 11,138,790, the 3Dsurface mesh model may have the following properties: all disjointedsurfaces are closed manifolds, appropriate supports are used to keep thedisjointed surfaces/volumes in place, appropriate supports are used tofacilitate 3D printing, and/or no surface volumes are hollow, such thatthe 3D surface mesh model is 3D printable. Moreover, 3D surface meshmodel generation module 118 may generate 3D surface mesh models of thepatient specific anatomical features including any correspondinglandmarks of the anatomical features, as described in further detailbelow.

Anatomical feature information extraction module 120 may be executed byprocessor 102 for extracting information from the 3D surface mesh modelgenerated by 3D surface mesh model generation module 118. For example,anatomical feature information extraction module 120 may extract one ormore specific anatomical features from the 3D surface mesh modelrepresenting the patient specific anatomical features within the medicalimages, based on the selected pathology indicated in the metadatareceived by image receiver module 112. Alternatively, platform 100 mayreceive information indicative of a selected pathology associated withthe medical images directly from the user via user interface 108, e.g.,along with patient demographic data and medical history. Accordingly, ifa specific pathology is known for a given patient, anatomical featureinformation extraction module 120 may automatically extract the 3Dsurface mesh model of the specific anatomical feature including thepathology from the 3D surface mesh model generated by 3D surface meshmodel generation module 118.

Physiological information generation module 122 may be executed byprocessor 102 for generating physiological information associated withthe selected pathology for the 3D surface mesh model based on theinformation extracted by anatomical feature information extractionmodule 120. For example, based on the selected pathology, physiologicalinformation generation module 122 may perform calculations to determinephysiological measurements relevant to the diagnosis and/or treatment ofthe pathology, e.g., by providing a list of medical devices appropriateto treat the pathology and/or treatment options based on measurements ofthe anatomical feature and patient demographic data. The list of medicaldevices and/or treatment options may be extracted from a medical devicedatabase or a surgical implement database by physiological informationgeneration module 122. As described in further detail below withreference to FIGS. 3A to 24F, the physiological measurements associatedwith the selected pathologies determined by physiological informationgeneration module 122 may include, but are not limited to, volume,cross-sectional area, diameter, centerline, surface, density, thickness,tortuosity, fracture size and location, blood clots, occlusions, andrate of growth over time of the anatomical feature and/or correspondinglandmark. Moreover, physiological information generated by physiologicalinformation generation module 122 may be used to generate heat maps tofacilitate visual observation of the physiological measurements of thepatient specific anatomical feature.

Referring now to FIG. 2 , exemplary method 200 for multi-schema analysisof patient specific anatomical features from medical images usingplatform 100 is provided. At step 202, medical images and metadataassociated with the medical images indicative of a selected pathologymay be received image receiver module 112. As described above,information indicative of the selected pathology may be directlyreceived via user input along with patient demographic data. At step204, segmentation module 114 may automatically process the medicalimages using a segmentation algorithm to label pixels of the medicalimages and to generate scores indicative of a likelihood that the pixelswere labeled correctly. For example, the segmentation algorithm may useone or a combination of various machine learning based imagesegmentation techniques, trained with a knowledge dataset of pre-labeledmedical images, to label pixels of the medical images.

At step 206, anatomical feature identification module 116 may grouptogether pixels labeled at step 204 based on similarities, and use ananatomical feature identification algorithm to probabilistically matchassociated groups of the labeled pixels against an anatomical knowledgedataset to classify one or more patient specific anatomical featureswithin the medical images. At step 208, 3D surface mesh model generationmodule 118 may generate a 3D surface mesh model defining a surface ofthe one or more classified patient specific anatomical features withinthe medical images. At step 210, anatomical feature informationextraction module 120 may extract information from the 3D surface meshmodel based on the selected pathology, and physiological informationgeneration module 122 may generate physiological information associatedwith the selected pathology for the 3D surface mesh model based on theextracted information. The physiological information generated isdescribed in further detail below with reference to FIGS. 3A to 24F.

Referring now to FIG. 3 , exemplary method 300 for generating volumemeasurements of a patient specific anatomical feature is provided. Asdescribed above with regard to step 210 of method 200 for multi-schemaanalysis of patient specific anatomical features from medical images ofFIG. 2 , physiological information, e.g., volume measurements of thepatient specific anatomical feature associated with the selectedpathology, may be generated from the generated 3D surface mesh model.For example, at step 302, a specific anatomical feature may be isolatedfrom the patient specific anatomical features within the medical imagesbased on the selected pathology, e.g., as indicated by the metadataassociated with the medical images, for further analysis, such that a 3Dsurface mesh model of the isolated anatomical feature may be extractedfrom the 3D surface mesh model of the patient specific anatomicalfeatures and recorded. Accordingly, only the anatomical feature(s)comprising the pathology may be further analyzed to generatephysiological information associated with the pathology.

At step 304, a start point and an end point of the isolated anatomicalfeature is determined, e.g., at opposite ends of the isolated anatomicalfeature. For example, the start and end points may be determined via amachine learning algorithm that explores the anatomical knowledgedataset to derive the start and end points of the isolated anatomicalfeature. At step 306, a predetermined step size may be determined, suchthat slices may be taken at regular intervals defined by thepredetermined step size along an axis of the isolated anatomicalfeature. For example, the axis may the centerline of the isolatedanatomical feature determined based on a directional vector extendingfrom the start point to the end point, as described in further detailbelow. Accordingly, a slice of the isolated anatomical feature may betaken at each interval perpendicular to the direction of travel alongthe centerline, beginning from the start point and in the direction ofthe end point.

At step 308, using standard computational functions, the cross-sectionalarea at each slice of the isolated anatomical feature may be calculated,as defined by the perimeter of the isolated anatomical feature, as shownin FIG. 4A. For example, the cross-sectional area of the automaticallysegmented labels for specific portions of anatomy, e.g., the mitral oraortic valve anatomy, may be calculated based on a derivative of thelargest two cross sections of the anatomy, e.g., using A×B×π. In thecase of an aneurysm, this data may be used to provide surgeons aneck-to-dome ratio automatically for the anatomy. FIG. 4A illustratesthree slices along an isolated anatomical feature, e.g., an aorta whenthe associated pathology is an aneurysm, for which cross-sectional areashave been calculated and displayed over the 3D surface mesh model of theaorta. FIG. 4B illustratively shows how slices may be taken along anaxis of a complex structure for purposes of calculating cross-sectionalareas thereof.

Referring again to FIG. 3 , at step 310, the 3D volume between eachadjacent slices may be extrapolated based on the cross-sectional areasof the isolated anatomical feature at adjacent slices, such that theoverall volume of the isolated anatomical structure may be determinedbased on the extrapolated 3D volumes, e.g., by taking the sum of theextrapolated 3D volumes. Alternatively, the volume of the automaticallysegmented labels for specific portions of an isolated anatomicalfeature, e.g., the left atrial appendage of the heart, may be calculatedbased on the number of voxels within the semantically labeled portion ofanatomy, such that the volume may be displayed to the user forassessment.

Referring now to FIG. 5 , exemplary method 500 for generating centerlinemeasurements of a patient specific anatomical feature is provided. Asdescribed above with regard to step 210 of method 200 for multi-schemaanalysis of patient specific anatomical features from medical images ofFIG. 2 , physiological information, e.g., centerline measurements of thepatient specific anatomical feature associated with the selectedpathology, may be generated from the generated 3D surface mesh model.For example, at step 502, a specific anatomical feature may be isolatedfrom the patient specific anatomical features within the medical imagesbased on the selected pathology, as described above, such that a 3Dsurface mesh model of the isolated anatomical feature may be extractedfrom the 3D surface mesh model of the patient specific anatomicalfeatures and recorded.

At step 504, a start point and an end point of the isolated anatomicalfeature is determined, e.g., at opposite ends of the isolated anatomicalfeature, such that a directional vector may be determined that extendsfrom the start point toward to end point. For example, the start and endpoints may be determined via a machine learning algorithm that exploresthe anatomical knowledge dataset to derive the start and end points ofthe isolated anatomical feature. The start and end points further may beclose to a bounding box of the 3D surface mesh model, and on a commonplane. Moreover, an initial direction of travel may be determinedconsistent with the directional vector extending from the start point tothe end point.

At step 506, a predetermined step size may be determined, such that acutting plane may be established at regular intervals defined by thepredetermined step size along an axis of the isolated anatomicalfeature. The cutting plane at each interval may be perpendicular to thedirection of travel associated with the interval. For example, theinitial cutting plane may be perpendicular to the initial direction oftravel based on the directional vector extending from the start point tothe end point. Moreover, multiple rays, e.g., three rays, may be raycastin multiple predefined directions along the cutting plane at eachinterval, perpendicular to the direction of travel and radiallyoutwardly toward the perimeter of the isolated anatomical feature, suchthat the position of the intersections of the rays cast and the 3Dsurface mesh model may be determined. For example, as shown in FIG. 6 ,in a direction of travel from start point SP toward end point EP, thefirst set of three rays cast may intersect the 3D surface mesh model ofthe isolated anatomical feature, e.g., vessel V, at points 602 a, 602 b,602 c. At step 506, if the point from which the rays are cast aredetermined to be outside of the 3D surface mesh model, the point may bemoved to within the 3D surface mesh model.

At step 508, the center point, e.g., CP1, of the isolated anatomicalfeature within the cutting plane at each interval may be determined,e.g., by triangulating the distances between each of the intersectionpoints, e.g., points 602 a, 602 b, 602 c, of the isolated anatomicalfeature. At step 510, a new direction of travel may be determined ateach interval based on a directional vector extending from the previouscenter point of the previous interval and the current center point. Forexample, in FIG. 6 , the new direction of travel at the first intervalmay be consistent with a directional vector extending from start pointSP to center point CP1. If the isolated anatomical feature is a branchedvessel, steps 506 to 510 may be repeated through both branches of thevessel, thereby generating a centerline for each branch of the 3Dsurface mesh model of the vessel.

Method 500 may repeat steps 506 to 510 until end point EP is reached.For example, as shown in FIG. 6 , at the second interval, three rays maybe cast along a cutting plane perpendicular to the direction of travelthat is defined by the directional vector extending from start point SPto center point CP1. The distances between intersection points 604 a,604 b, 604 c of the rays cast and the 3D surface mesh model may betriangulated to determine center point CP2 at the second interval. Theprevious direction of travel may then be adjusted to a new direction oftravel defined by the directional vector extending from center point CP1to center point CP2. Similarly, at the third interval, three rays may becast along a cutting plane perpendicular to the direction of travel thatis defined by the directional vector extending from center point CP1 tocenter point CP2. The distances between intersection points 606 a, 606b, 606 c of the rays cast and the 3D surface mesh model may betriangulated to determine center point CP3 at the third interval. Theprevious direction of travel may then be adjusted to a new direction oftravel defined by the directional vector extending from center point CP2to center point CP3. As described above, steps 510 to 512 may berepeated until end point EP is reached to thereby determine a series ofcenter points CP along an axis of the isolated anatomical feature, asshown in FIG. 7A. Accordingly, as described above, the point from whichthe rays are cast will be outside of the 3D surface mesh model beyondend point EP, such that the point cannot be returned to within the 3Dsurface mesh model, thereby indicating an end of the centerline of theisolated anatomical feature.

At step 512, the centerline of the isolated anatomical feature may bedetermined based on the totality of center points, e.g., CP1, CP2, CP3 .. . CPn. For example, the centerline may be a line drawn through all ofthe calculated center points of the isolated anatomical feature, asshown in FIG. 6 . FIG. 7B illustrates centerline CL of an isolatedanatomical feature as a line drawn through all of center points CP ofFIG. 7A. Accordingly, as shown in FIG. 7C, the overall length ofcenterline CL of the isolated anatomical feature may be determined. FIG.7D illustrates the 3D surface mesh model of the isolated anatomicalfeature of FIGS. 7A to 7C across the original medical images.

Referring now to FIG. 8 , method 500 may be used to determine thecenterlines of a vast network of patient specific anatomical features.For example, FIG. 8A illustrates the start and end points determined fora 3D surface mesh model of an isolated anatomical feature. FIG. 8Billustrates centerline CL determined for an isolated anatomical featuremapped to the original medical image. FIG. 8C illustrates centerlines CLfor an anatomical feature comprising a plurality of vessels, and FIG. 8Dillustrates centerlines CL for an anatomical feature comprising a vastnetwork of vessels.

Referring now to FIG. 9 , exemplary method 900 for generating surfacelength measurements of a patient specific anatomical feature isprovided. As described above with regard to step 210 of method 200 formulti-schema analysis of patient specific anatomical features frommedical images of FIG. 2 , physiological information, e.g., surfacelength measurements of the patient specific anatomical featureassociated with the selected pathology, may be generated from thegenerated 3D surface mesh model. For example, at step 902, a specificanatomical feature may be isolated from the patient specific anatomicalfeatures within the medical images based on the selected pathology, asdescribed above, such that a 3D surface mesh model of the isolatedanatomical feature may be extracted from the 3D surface mesh model ofthe patient specific anatomical features and recorded.

At step 904, the centerline of the isolated anatomical feature may bedetermined, e.g., via method 500 described above. At step 906, a startpoint and an end point of the isolated anatomical feature may bedetermined, e.g., at opposite ends of the isolated anatomical feature.At step 908, a predetermined step size may be determined, such that acutting plane may be established at regular intervals defined by thepredetermined step size along an axis of the isolated anatomicalfeature. As shown in FIG. 10 , the cutting planes, e.g., P1, P2, at eachinterval of the isolated anatomical feature, e.g., vessel V, may beperpendicular to the direction of travel associated with the interval,e.g., the direction of travel of the centerline at the interval asdescribed above, and may include the center point along centerline CL,e.g., CP1, CP2, at the respective interval and a point along directionalvector DV extending from start point SP to end point EP.

At step 910, a ray, e.g., rays R1, R2, may be cast along each cuttingplane, e.g., cutting plane P1, P2, at each interval from the respectivecenter point, e.g., center points CP1, CP2, radially outwardly towardthe 3D surface mesh model, such that the position of the intersectionbetween the rays and the 3D surface mesh model are recorded, e.g.,intersection points D1, D2. Step 10 may be repeated at each predefinedinterval to determine a series of intersection points along the surfacetopology of the 3D surface mesh model. At step 912, the overall lengthof a line extending across the surface of the 3D surface mesh model ofthe isolated anatomical feature, as defined by the intersection pointsdetermined at step 910, may be calculated based on the determinedintersection points. FIG. 11 illustrates surface line SL extendingacross the surface topology of a 3D surface mesh model of an isolatedanatomical feature.

For example, regarding cardiac image segmentation, once the automatedsegmentation has been completed, a 3D surface mesh model of the heartsurrounding vessels will be created. This 3D data may then beautomatically analyzed to assess specific lengths pertaining to thelandmarks of the heart which may include, but are not limited to:atrium; ventricle; aorta; vena cava; mitral valve; pulmonary valve;aortic valve; tricuspid valve; myocardium; coronary arteries; leftatrial appendages.

Referring now to FIG. 12 , exemplary method 1200 for generating a heatmap of a patient specific anatomical feature based on volume isprovided. As described above, the cross-sectional area of the isolatedanatomical feature at predefined intervals along an axis of the isolatedanatomical feature may be determined, such that a heat map of the 3Dsurface mesh model may be generated based on cross-sectional areas ofthe 3D surface mesh model along the axis of the isolated anatomicalfeature. For example, at step 1202, a specific anatomical feature may beisolated from the patient specific anatomical features within themedical images based on the selected pathology, as described above, suchthat a 3D surface mesh model of the isolated anatomical feature may beextracted from the 3D surface mesh model of the patient specificanatomical features and recorded. At step 1204, a start point and an endpoint of the isolated anatomical feature is determined, and an initialdirection of travel may be determined consistent with the directionalvector extending from the start point to the end point. At step 1206,the centerline of the isolated anatomical feature may be determined,e.g., via method 500 described above.

At step 1208, a predetermined step size may be determined, such thatslices may be taken at regular intervals defined by the predeterminedstep size along the centerline of the isolated anatomical feature.Accordingly, a slice of the isolated anatomical feature may be taken ateach interval perpendicular to the direction of travel along thecenterline. At step 1210, using standard computational functions, thecross-sectional area at each slice of the isolated anatomical featuremay be calculated, as defined by the perimeter of the isolatedanatomical feature. At step 1210, a heat map may be generated based onthe cross-sectional areas at each slice of the 3D surface mesh model,thereby visually indicating the change in volume throughout the isolatedanatomical feature, as shown in FIGS. 13A and 13B.

Referring now to FIG. 14 , exemplary method 1400 for generating a heatmap of a patient specific anatomical feature based on tortuosity isprovided. As described above, the direction of travel at predefinedintervals of the centerline of the isolated anatomical feature may bedetermined, such that a heat map of the 3D surface mesh model may begenerated based on the magnitude of change of the direction of travelalong the axis of the isolated anatomical feature. For example, at step1402, a specific anatomical feature may be isolated from the patientspecific anatomical features within the medical images based on theselected pathology, as described above, such that a 3D surface meshmodel of the isolated anatomical feature may be extracted from the 3Dsurface mesh model of the patient specific anatomical features andrecorded. At step 1404, a start point and an end point of the isolatedanatomical feature is determined, and an initial direction of travel maybe determined consistent with the directional vector extending from thestart point to the end point. At step 1406, the centerline of theisolated anatomical feature may be determined, e.g., via method 500described above.

At step 1408, the direction of travel at predefined intervals of thecenterline of the isolated anatomical feature may be determined, e.g.,based on the directional vectors extending between adjacent centerpoints along the centerline as described above. At step 1410, themagnitude of change between the direction of travel of adjacentintervals may be determined. For example, the magnitude of change may becalculated using the directional vectors associated with the respectivedirections of travel at each interval. At step 1412, a heat map may begenerated based on the magnitudes of change between the direction oftravel of adjacent intervals along the axis of the 3D surface meshmodel, thereby visually indicating the tortuosity of the isolatedanatomical feature, as shown in FIG. 15 . Accordingly, the magnitude ofchange, e.g., angular changes, that are outputted from the analysis maybe cross-referenced with an existing knowledge database of knownclassification angular deviations, and displayed to the user. Thetortuosity value may be depicted as a total change in angle of thevessel and scored, e.g., 760 degrees rotation score.

Referring now to FIG. 16 , exemplary method 1600 for generating a 3Dsurface mesh model of a patient specific anatomical feature withidentified landmarks is provided. As described above with regard to FIG.2 , medical images, as shown in 1702 of FIG. 17A, may be automaticallyprocessed to identify patient specific anatomical features, as shown in1704 of FIG. 17A, such that a 3D surface mesh model of the classifiedpatient specific anatomical features within the medical images may begenerated. Method 1600 further identifies corresponding landmarks of thepatient specific anatomical features, e.g., a bone notch or heart valve,such that the landmarks may be depicted in the 3D surface mesh model.For example, prior to generation of the 3D surface mesh model based onthe classified patient specific anatomical features, at step 1602,information indicative of a specific anatomical feature may be isolatedfrom the data representing the patient specific anatomical featureswithin the medical images based on the selected pathology, as shown in1706 of FIG. 17A (anatomy delineation).

At step 1604, features of the isolated anatomical feature may beanalyzed with an anatomical feature dataset to identify one or morelandmarks of the isolated anatomical feature associated with theselected pathology. For example, the anatomical feature dataset mayinclude knowledge of anatomical landmarks, e.g., existing semanticallylabeled anatomical feature datasets, associated with various patientspecific anatomical features, such that the landmarks may be identifiedand individually labeled by establishing links between the classified,isolated anatomical feature and the anatomical feature dataset. At step1606, the identified, labeled landmarks may be associated with pixels ofthe original medical images, as shown in 1708 of FIG. 17A. At step 1608,a 3D surface mesh model of the isolated anatomical feature may begenerated depicting the identified landmarks mapped to the pixels of themedical images associated therewith, as shown in 1710 of FIG. 17A.

The identified anatomical landmarks are a meaningful point in apatient's anatomy that has significance to its form or function, such asorientation and insertion points for other anatomical features. Theidentified landmarks may help surgeons ensure the landmarks correspondto a specific portion of anatomy and ensure its proper function andorientation. The identified landmarks further may be utilized withinclinical practice as markers on anatomy to facilitate diagnosis and/ortreatment of a patient, e.g., as an initial reference for anatomicalguide fixation and trajectory planning. For example, specific anatomicallandmarks identified for each bone may be automatically detected, suchthat a guide may be generated for cutting and drilling of the bone.Thus, the identified anatomical landmarks may be used as inputs forclinical functions have significant benefits. For example, FIG. 17Billustrates the following identified landmarks: (A) fossa center, (B)trigonum, (C), inferior angle, (D) center of spine of scapula, mapped tothe isolated anatomical feature, e.g., a scapula for shoulderreplacement. Accordingly, the identified landmarks may serve as areference to provide guidance for cutting planes and drillingtrajectories within bones, as well as for device fixation in the bone.

Referring now to FIG. 18 , exemplary method 1800 for identifying medicaldevices and treatment options for a pathology is provided. For example,at step 1802, a specific anatomical feature may be isolated from thepatient specific anatomical features within the medical images based onthe selected pathology, as described above, such that a 3D surface meshmodel of the isolated anatomical feature may be extracted from the 3Dsurface mesh model of the patient specific anatomical features andrecorded, as shown in FIG. 19A. At step 1804, physiological parametersof the isolated anatomical feature may be analyzed, as described above,for example, to determine measurements such as volume, centerline,surface length, cross-sectional area, diameter, density, etc.

Based on the physiological parameters of the isolated anatomical featureas well as patient demographic data associated with the medical images,at step 1806, one or more medical devices and/or treatment options maybe identified from a medical device database having knowledge of variousmedical devices including their function and specifications and/or asurgical implement database having knowledge of pathology-specifictreatment options. For example, physiological parameters of the isolatedanatomical feature may indicate the size of a selected pathology, suchthat a specific sized medical device that is known to be used to treatthe selected pathology may be identified for use in treating thepathology. The identified medical devices may further be selected froman internal inventory, e.g., medical devices available or provided by aspecific hospital. The knowledge datasets described herein may furtherinclude knowledge of the combination of anatomy with non-organicmaterial, e.g., polymers, metals and ceramic, such that non-organicmaterial may also be auto-segmented. In addition, the knowledge datasetsmay include knowledge of medical devices which may be used as inputs forcreation of patient specific guides, e.g., knowledge of preexistingimplants for the correction of bony pathologies. For example, knowndimensions and variabilities of the devices may be used as inputs in thedevice's automated design. At step 1808, the identified medical devicesand/or treatment options may be displayed to the user, such that theuser may make an informed decisions regarding preoperative planning andtreatment, as shown in FIGS. 19B and 19C.

The ability to provide the automated segmentation opens up a number ofbeneficial pathology specific applications. For example, some specificpathologies/treatments that require higher volume 3D models (virtual orphysical) are listed in Table 1 below.

TABLE 1 Where Pathology How to treat C LAA - left atrial appendageOcclusion device (watchman) C Mitral valve regurgitation Mitral valvereplacement C Aortic valve regurgitation TAVI/TAVR - Transcatheteraortic valve implantation C Aortic aneurysms Patient specific stent IRAAA - Abdominal aortic Patient specific Stent aneurysms C Septal defects(ventricle Occlusion device or atrium) C Coronary heart diseaseArthrectomy/Angioplasty via catheter or coronary bypass N IschemicStroke Aspiration stent or no stent retrieval catheter N Hemorrhagicstroke Craniotomy N Neuro Aneurysm (ICA) Stent, coil or clip O BoneFractures Plates or Patient specific instrumentation O Primaryorthopaedic Revision instrument - Patient replacement failure (Hip,specific guide & Patient specific knee, pelvis) instrumentation O-OnOsteosarcoma Patient specific guide & Patient specific instrumentation OScoliosis Plates or Patient specific instrumentation O Osteoarthritis -hip Joint replacement, primary hip replacement Ortho instruments OOsteoarthritis - knee Joint replacement, primary knee replacement Orthoinstruments O Osteoarthritis- knee Joint replacement, primary shoulderreplacement Ortho instruments ON General oncology (lung, Resection orradiation of tumor liver, Kidney, skull base, mass brain) M Midfacedeformities Le Fort procedure- facial reconstruction with osteotomies GColon disease - Bowel Stoma and colostomy bag cancer, Crohn's disease,,colitis, diverticulitis G Prostate enlargement, bladder Urinary cathetercancer & urinary incontinence O Cruciate ligament/meniscus Kneereplacements tears C Aortic Stenosis Transcatheter aortic valvereplacement O—Ortho C—cardiac/cardiology N— Neuro G—General M—Max faxOn—Oncology IR—Interventional radiology

Referring now to FIG. 20 , exemplary method 2000 for detecting andclassifying a fracture of a patient specific anatomical feature isprovided. As described above with regard to FIG. 2 , medical images, asshown in FIG. 21A, may be automatically processed to identify patientspecific anatomical features, such that a 3D surface mesh model of theclassified patient specific anatomical features within the medicalimages may be generated. Method 1600 further detects/identifiescorresponding fractures of the patient specific anatomical features,e.g., in a bone such as the tibia, fibia, or medial malleolus, such thatthe fractures may be depicted in the 3D surface mesh model. For example,prior to generation of the 3D surface mesh model based on the classifiedpatient specific anatomical features, at step 2002, informationindicative of a specific anatomical feature may be isolated from thedata representing the patient specific anatomical features within themedical images based on the selected pathology, as shown in FIGS. 21Band 21C.

At step 2004, features of the isolated anatomical feature may beanalyzed with an anatomical feature dataset to identify one or morelandmarks of the isolated anatomical feature, e.g., a bone notch,associated with the selected pathology. As described above, theanatomical feature dataset may include knowledge of anatomical landmarksassociated with various patient specific anatomical features, such thatthe landmarks may be identified and individually labeled by establishinglinks between the classified, isolated anatomical feature and theanatomical feature dataset. At step 2006, features of the identifiedlandmark may be analyzed with an reference fracture database to identifyone or more fractures of the identified landmark of the isolatedanatomical feature associated with the selected pathology. The referencefracture database may include knowledge of various fractures, e.g.,existing semantically labeled reference fracture datasets, associatedwith various patient specific anatomical features, such that thefractures may be identified and individually labeled by establishinglinks between the classified, isolated anatomical feature and theanatomical feature dataset. At step 2008, a 3D surface mesh model of theisolated anatomical feature may be generated depicting the identifiedlandmarks and detected fracture F, as shown in FIG. 21D. Moreover, atstep 2010, the 3D surface mesh model may be matched against thereference fracture database to classify the fracture type.

Referring now to FIG. 22 , exemplary method 2200 for trackingprogression of a pathology over time is provided. For example, at step2202, a specific anatomical feature may be isolated from the patientspecific anatomical features within the medical images based on theselected pathology, as described above, such that a 3D surface meshmodel of the isolated anatomical feature may be extracted from the 3Dsurface mesh model of the patient specific anatomical features andrecorded. At step 2204, physiological parameters of the isolatedanatomical feature may be analyzed, as described above, for example, todetermine measurements such as volume, centerline, surface length,cross-sectional area, diameter, density, etc.

For example, once the automated segmentation has been completed, a 3Dsurface mesh model of the aneurysm and vascular anatomy may begenerated. This 3D data may then be automatically analyzed to assessspecific lengths pertaining to the aneurysm morphology, which mayinclude, but are not limited to measurements of the aneurysm neck,diameter measurements of the aneurysm at maximum distances, andmeasurements of center points of the superior and inferior aneurysmnecks.

At step 2206, the analyzed physiological parameters of the isolatedanatomical feature may be timestamped and recorded, such that over time,there is a chronological record of the physiological parameters for aspecific patient. At step 2208, changes between the recorded/timestampedphysiological parameters over time may be calculated to indicate, e.g.,progression and prognosis of the selected pathology. For example, FIGS.23A to 23F illustratively show growth of various aneurysms over time,leading to eventual rupture. Optionally, at step 2210, a heat map may begenerated to visually depict the changes between therecorded/timestamped physiological parameters over time, as shown inFIGS. 24A to 24F.

Referring now to FIG. 25 , exemplary method 2500 for semantic volumerendering is provided. A single medical image 2602 of a stack of medicalimages 2604 is shown in FIG. 26 . Volume rendering is an importantsolution that is adopted by medical professionals globally to visualizemedical imaging datasets in 3D space. They work by mapping pixelcharacteristics such as specific color, intensity, or opacity tospecific voxels within the 3D scene. There are deficiencies associatedwith this method of imaging whereby overlapping and deep structures arenot easily visualized in detail. Thus, to cure these deficiencies,method 2500 generates 3D surface mesh models of separate anatomicalfeatures, such that physiological parameters of the separate anatomicalfeatures may be analyzed.

For example, the results of the automatic image segmentation may takethe form of a series of binary pixel arrays contained in medical images,e.g., DICOM files. When assembled into a volume, the binary pixel arraysmay be used to mask the areas of the source pixel volume that are notrelevant to the identified anatomy. The remaining Hounsfield valuevolume may then be rendered using standard volume rendering techniqueswith the color transfer function, such that pixel intensity may bedetermined based on the Hounsfield values. Moreover, the length of theanatomical feature, e.g., a vessel, may be calculated based on theoutput from the automated segmentation algorithm and subsequent 3Dreconstruction. The data extracted from the 3D reconstruction may thenbe automatically analyzed to output a length from one specificanatomical landmark or abnormality to another, e.g., the length from theaortic arch to the thrombus in the case of a stroke. The measurement inthe case of a vessel may be calculated by creating a center point on across section of the vessel, and extrapolated the center points throughthe vessel and joining the center points to create a centerline of theanatomy. This centerline may then be automatically measured andoutputted to the user as a length value.

For example, at step 2502, the classified patient specific anatomicalfeatures generated using the segmentation algorithm described above aredelineated into binary labels, e.g., bone/not bone, vessel/not vessel,organ/not organ, etc. At step 2504, the binary labels are separated intoseparate anatomical features, e.g., myocardium of heart, aorta, coronaryarteries, etc. At step 2506, the separate anatomical features are mappedto the original medical images, such that only the original grey scalevalues or Hounsfield units for the separate anatomical features areshown in the medical images, as shown in 2606 and 2608 of FIG. 26 andFIG. 27B, and the background may be removed from the medical images asshown in 2610 and 2612 of FIG. 26 and FIG. 27C, leaving visible only theseparate anatomical features depicted in the original grey scale valuesor Hounsfield units.

At step 2508, a 3D surface mesh model of the separate anatomicalfeatures may be generated. The 3D surface mesh model may define asurface of the separate anatomical features, as shown in 2614 of FIG. 26. Additionally or alternatively, the specific colors of transparencyvalues may be mapped to labeled 3D surface mesh model to generate avolumetric render, as shown in 2616 of FIG. 26 and FIG. 27D. Forexample, a color map of the pixel intensities may be mapped directly tothe 3D voxel intensities within only the segmentation to allow forspecific volumetric visualization of the isolated anatomical feature.The voxels may be given a specific color automatically depending on theintensities of the original image, which may be indicative of normalblood flow or lack thereof. The ability to color specific regions ofinterest such as a clot, break, or anatomy, allows for greater insightinto a specific pathology of a region.

As shown in FIG. 27D, the 3D volumetric render may indicate the presenceof a clot/occlusion. This data may then be rendered on an end-userapplication such that the 3D volumetric render may be rotated orotherwise manipulated and viewed. This data also may be used to indicateto the user if calcification is present from, e.g., a grouping of highintensity pixels, and further may provide a calcification “score” byindicating the percent of the clot or occlusion that is representativeof the calcified structure. For example, predictions of theocclusions/calcifications may be made and applied as a mask on theoriginal medical image, such that the background portions of the medicalimage may be removed, as shown in 2802 of FIG. 28A. Accordingly, a 3Dsurface mesh model may be generated that takes into account the pixelintensity of the various materials, as shown in 2804, 2806, and 2808 ofFIG. 28A. As shown in FIG. 28B, the size of occlusion O depicted invessel V of the 3D volumetric render may be measured, e.g., forassisting in the diagnosis and treatment for stroke patients.

The 3D volumetric render may be set by the user or automatically derivedto visualize specific features by referencing the anatomical featuresdepicted in the volumetric render, such as clots within vascularstructures, coronary arteries, neuro vessels, thereby indicating apotential stroke. Accordingly, the medical images may be automaticallysegmented and reconstructed, e.g., by utilizing CTA's/XA/NM vesselimaging for the patient, to create a 3D representation of both a vesseland associated occlusions using machine learning from a semanticallylabeled 3D anatomical knowledge dataset that may be easily viewed on amobile device or similar platform.

Once the 3D surface mesh model is generated from the automatedsegmentation, it will be possible to generate a number of measurementsabout the anatomy or pathology in the medical scan. Moreover, thescaling information along with reference points permits placement of thepatient specific anatomical features within a physical scene. At themost simple level, physical measurements may be generated of the mesh,or any sub-mesh, or otherwise delineated region in the physical scene,which may include: length, breadth, height, angles, curvature,tortuosity of a mesh, etc. Given a filled structure, measurements mayalso be made of the volume, surface area, and diameter.

Derived properties of the materials to be segmented may also bemeasured. At a basic level, these may include thickness of the material(blood vessel or bone), and a known derivation from the normal (patientor general), which may permit generation of predictions about, e.g., thelikely pressure required to break the material, or simply supply avisualization of the thickness and stress lines. Visualization of any ofthe above mentioned measurements provides great value as any moreinformation available to the surgeon would be helpful in thedetermination of the best course of action for treatment, and wouldprovide the ability to give an accurate analysis of the diagnosis. Thismay be achieved through a simple overlay of the derived variable overthe mesh or by providing the data for additional analysis of theinput/desired attribute.

Aside from the determining the structure of a patient specificanatomical feature as described above, an extracted polygonal model mayfurther provide a convenient basis for determining numerous usefulmeasurements that would otherwise be difficult to ascertain fromvolumetric pixel data alone, e.g., bone and vessel dimensions, angle andtortuosity differentials and relative scales, density etc. Normallydetermining these measurements would require careful manual assessmentof a mesh in order to identify areas of interest and meaningfulreference points. However, the exploratory geometric algorithmsdescribed herein provides a reliable automated alternative. For example,the following pseudocode outlines how vessel length, diameter andcurvature information may be automatically collected without humanintervention:

getVesselInfo (mesh) {  -get Bounding Box of input mesh  -get minimumand maximum coordinates along each axis  -any vertices existing at theseextreme points can be presumed to form part of the circular opening of avessel  -build circular/elliptical entry points by clustering previouslyidentified  extreme vertices  -get centre points of vessel openings -determine inward direction of vessels from volume  -for each entrypoint centre   -while ray cast hasn't collided with planes defined byvessel entry   points    -create new measurement line    -raycast atdifferent equidistant angles    -take longest distance    -advance alongdistance line    -centre in vessel diameter by calculating centre ofsmallest    diameter line    -(save diameter value for determiningthickness differential later)    -add new location to measurement line   -in the event of multiple peaks in the array of distances    -foreach branch continue march   -remove exit point from entrypointlist  -return resulting directional paths

Referring now to FIG. 29 , exemplary method 2900 for analyzingphysiological parameters of separate anatomical features is provided.Some of the steps of method 2900 may be further elaborated by referringto FIGS. 30A to 30E, which depict a 2D example of a cross-section of avessel with branching paths. FIG. 30A illustrates branched vessel V. Atstep 2902, planes P1, P2, P3 may be built at the entry points of vesselV, defined by the boundaries of the volume of vessel V, as shown in FIG.30B. At step 2904, center points C1, C2, C3 of entry planes P1, P2, P3,respectively, may be calculated, as shown in FIG. 30C. As shown in FIG.30C, multiples ray may be raycast from center point C3 into thestructure of vessel V to determine the longest unobstructed path withinvessel V. Because of the branching paths of vessel V, there are two peakpoints PP1, PP2 depicted in FIG. 30C. This may be determined byassessing the number of inflection points in the graph of distancevalues. Having determined that there are numerous paths forward at thispoint in the algorithm, each branch may be assessed individually bybranching off the control flow.

At step 2906, the entire structure of vessel V is marched through untilthe rays cast at each point along lines L1, L2 intersect with entryplanes P2 and P3, respectively, as shown in FIG. 30D, resulting in aseries of vertices charting each of the paths through vessel V. At step2908, a best fit spline line may be constructed through the verticesalong lines L1, L2, as shown in FIG. 30E, such that diametricmeasurements may be taken at each point along lines L1, L2 to therebyprovide a complete representation of vessel V, of whichslope/tortuosity, diameter, internal volume, etc. may be determined.

Moreover, working from the pseudocode described above, the presence of apathology such as an aneurysm would result in the search point gettingstuck in a loop. Whenever the points of the measurement line begin torepeatedly change direction. The algorithm may break out of the searchloop and presume that an aneurysm has been entered. Accordingly, thephysiological measurements of the aneurysm may be determined, e.g., bydetermining points around the entry to the aneurysm, building entryplane to the aneurysm, determining the center point of the entry plane,and raycasting into the aneurysm structure to determine the most distantpoint, and when the max distance has been determined, building a linebetween the entry plane and max distant point, and begin checkingperpendicular distances by raycasting.

The results of the segmentation may be quantified by, e.g., measuringthe density of a segmented area, identifying the proximity to otherpieces of anatomy, and identifying and delineating boundaries,especially with regard to oncology. Once a region has been identifiedand delineated within the physical scene, statements about the regionmay be made in relation to other structures within the scene. Forexample, delineating tumor boundaries and understanding their distancefrom key structures in the anatomical neighborhood would be useful tooncologists. Moreover, the density of a given structure would provideclinically relevant information, e.g., in the case of oncology, it wouldprovide insight into hypoxia within the tumor, and in the case of ablood clot, it would allow insight into how the clot could be treated.

The ability to measure the density and thickness of an anatomical regionwould allow the ability to provide guidance on, e.g., screw selection intrauma applications or catheter diameter in vascular applications.Moreover, the ability to measure the diameters along an anatomicalfeature would allow the diameter measurements to be cross-referencedwith a medical device database to indicate to the surgeon the best sizeddevice for that patient.

The machine learning based algorithms described herein may be trainedand predicted on the axial axis, which is typically the axis that themedical scans are carried out in. A modification to the machine learningbased algorithm may involve changing the prediction function, andanother modification may involve changing the training and theprediction function. For example, a modification to the machine learningbased algorithm may include making predictions in all three axis andthen merging the results. This approach would work best where the voxelsare isotropic, as in the case with the rimasys data. The merging of thepredictions may follow a number of different strategies, for example,taking an average (mean) of the three results for a given pixel/voxel,or more complex solutions such as taking a weighted average of an axialslice plus the others. Alternatively, it would be possible to switch toa different primary axis, e.g., switching from an axial axis to asagittal.

Training the algorithm on all three axes may take advantage of theadditional information from the different axes. Thus, an axial inferencemodel, a sagittal inference model, and a coronal model may be trained.As described above, the results of all three predictions may be combinedwith a simple merging strategy. However, preferably, either the outputlayer of the three models may be combined in a larger network or anensemble model may be created that combines their results.

As described in U.S. Patent Appl. Pub. No. 2021/0335041, the algorithmmay to work natively in 3D, which may be very expensive from a memoryallocation point of view. One other approach to mitigate thisrestriction would be to consider a cube rather than a slice at a time.The advantage of this approach is that it may be possible to take intoconsideration the more pertinent and immediate context in the training,such that instead of considering a large thick slab, the algorithm istrained on small cubes of volume, which are slid over the entire volume.

The sandwich approach described in U.S. Patent Appl. Pub. No.2021/0335041 may be extended to incorporate a larger number of slices,may also more explicitly incorporate the pixels from the surroundingslices in the model. For example, instead of using additional channelsin the image, multiple channels, e.g., three channels, of most imageformats may be leveraged to achieve this compression. By making thesurrounding images into full images, the number of surrounding images ina scan may be generically increased. As the size of GPUs increase, thenumber of surrounding images in a scan may also be increased. Moreover,

The algorithm may implement a version of D-Unet which takes into accountthe 3D contextual information (via 3D convolution kernels), and theamount of slices the model analyzes at a time may be increased toprovide the algorithm much more spatial context. This architectureupgrade together with improvements to the loss functions and access tomore data has resulted in increasingly better segmentation models.

Moreover, the methods described herein further may utilize an Euclideandistance weight approach to influence the loss component in the machinelearning model training process. This approach helps guide the learningprocess to focus on areas of greater importance. For example, inorthopedics segmentation, the most difficult errors to detect/find andfix are small connections between bones that are very close to eachother; whereas, small holes on the inside of the bones are more simpleto correct. FIG. 31 illustrates weight masks generated with theEuclidean distance weight approach, as well as their effect on the lossfunction, e.g., categorical cross entropy.

A multi-schema approach to ground truth dataset for training isprovided. Specifically, there are many different segmentation labelingschemas that may be used to adapt the training labels depending on thegoal of the model to be trained. For example, as it may be verydifficult to define the inner materials of trauma bones, they aregenerally segmented as hollow, and thus the predictions from a traumamodel trained on hollow bone labels are much easier to work with, asshown in Table 2 below.

TABLE 2 Bone Segmentation Labelling Schemas Original Solid Bone LabelsMeaning Outer Bone Hollow Bone Solid Bone Only 0 Background 0(background) 0 (background) 0 (background) 0 (background) 1 External 0(background) 0 (background) 0 (background) 0 (background) 2 Outertrabecular 1 (bone) 1 (bone) 1 (bone) 1 (bone) 3 Inner trabecular 0(background) 0 (background) 1 (bone) 1 (bone) 4 Outer cortical 1 (bone)1 (bone) 1 (bone) 1 (bone) 5 Inner cortical 0 (background) 1 (bone) 1(bone) 1 (bone) 6 Outer marrow 1 (bone) 1 (bone) 1 (bone) 1 (bone) 7Inner marrow 0 (background) 0 (background) 1 (bone) 1 (bone) 8 Artifact2 (artifact) 2 (artifact) 2 (artifact) 0 (background) 9 Air 0(background) 0 (background) 0 (background) 0 (background)

FIG. 32 illustrates various segmentations of bone within medical imagesusing the multi-schema approach to ground truth data for trainingpurposes, as described above. Similarly, Table 3 illustrates cardiacsegmentation labeling schemas used with the multi-schema approach toground truth data.

TABLE 3 Cardiac Segmentation Labelling Schemas Original Labels MeaningCardiac Cardiac Only 0 Background 0 (background) 0 (background) 1External 0 (background) 0 (background) 2 Blood-flow 1 (blood-flow) 1(blood-flow) 3 Myocardium 2 (myocardium) 2 (myocardium) 4 Artifact 3(artifact) 0 (background) 5 Calcification 4 (calcification) 1(blood-flow)

FIG. 33 illustrates various segmentations of a myocardium within medicalimages of ground truth data for training purposes.

These same techniques for adapting label schemas may be used to definenormal versus pathological tissues, or lack of tissue in some examples,which will allow semantic segmentation of a pathology as a region ofinterest, and further allow pathology specific workflows to beautomatically started. Moreover, the multi-schema approach of the usingmultiple labels to differentiate anatomies and pathologies may be usedto semantically label each anatomical feature of the human body.Examples of various schema labels may include, but are not limited to:Nasal; Lacrimal; Inferior Nasal Concha; Maxiallary; Zygomatic; Temporal;Palatine; Parietal; Malleus; Incus; Stapes; Frontal; Ethmoid; Vomer;Sphenoid; Mandible; Occipital; Rib 1; Rib 2; Rib 3; Rib 4; Rib 5; Rib 6;Rib 7; Rib 8 (False); Rib 9 (False); Rib 10 (False); Rib 11 (Floating);Rib 12 (Floating); Hyoid; Sternum; Cervical Vertebrae 1 (atlas); C2(axis); C3; C4; C5; C6; C7; Thoracic Vertebrae 1; T2; T3; T4; T5; T6;T7; T8; T9; T10; T11; T12; Lumbar Vertebrae 1; L2; L3; L4; L5; Sacrum;Coccyx; Scapula; Clavicle; Humerus; Radius; Ulna; Scaphoid; Lunate;Triquetrum; Pisiform; Hamate; Capitate; Trapezoid; Trapezium; Metacarpal1; Proximal Phalange 1; Distal Phalange 1; Metacarpal 2; ProximalPhalange 2; Middle Phalange 2; Distal Phalange 2; Metacarpal 3; ProximalPhalange 3; Middle Phalange 3; Distal Phalange 3; Metacarpal 4; ProximalPhalange 4; Middle Phalange 4; Distal Phalange 4; Metacarpal 5; ProximalPhalange 5; Middle Phalange 5; Distal Phalange 5; Hip (Ilium, Ischium,Pubis); Femur; Patella; Tibia; Fibula; Talus; Calcaneus; Navicular;Medial Cuneiform; Middle Cuneiform; Lateral Cuneiform; Cuboid;Metatarsal 1; Proximal Phalange 1; Distal Phalange 1; Metatarsal 2;Proximal Phalange 2; Middle Phalange 2; Distal Phalange 2; Metatarsal 3;Proximal Phalange 3; Middle Phalange 3; Distal Phalange 3; Metatarsal 4;Proximal Phalange 4; Middle Phalange 4; Distal Phalange 4; Metatarsal 5;Proximal Phalange 5; Middle Phalange 5; Distal Phalange 5; Circle ofWillis; Anterior Cerebral Artery; Middle Cerebral Artery; PosteriorCerebral Artery; Lenticulostriate Arteries; brachiocephalic artery;right common carotid; right subclavian artery; vertebral artery; basilarartery; Posterior cerebral artery; posterior cerebral artery; posteriorcommunicating artery; left common carotid artery; internal carotidartery (ICA); external carotid artery (ECA); left subclavian artery;right subclavian artery; internal thoracic artery; thyrocervical trunk;costocervical trunk; left subclavian artery; aorta; Vena Cava; axilla;axillary artery; brachial artery; radial artery; ulnar artery;descending aorta; thoracic aorta; abdominal aorta; hypogastric artery;external iliac artery; femoral artery; popliteal artery; anterior tibialartery; arteria dorsalis pedis; posterior tibial artery; tricuspidvalve; pulmonary valve; mitral valve; aortic valve; Right Ventricle;Left ventricle; Right atrium; Left atrium; Liver; Kidney; Spleen; Bowel;Prostate; Cerebrum; Brainstem; Cerebellum; Pons; Medulla; Spinal cord;Frontal lobe; Parietal lobe; Occipital lobe; Temporal lobe; Rightcoronary artery; left main coronary; left anterior descending; leftcircumflex artery.

Hybrid data labeling for reinforced learning is provided. With amajority of machine learning models, creating a large corpus of data totrain on is essential. With regard to segmentation algorithms forlabeling DICOMS, as described herein, the ability to create largeamounts of data for robust algorithms is limited by the resources ofskilled engineers or imaging specialists. By utilizing the initialresults of segmentation algorithms, the methods described herein mayspeed up the time it takes to create a large dataset. For example:

-   Time to segment a single image (no automation)=10 seconds;-   Assumption for robust algorithms—100,000 labeled images;-   100,000 images segmented sequentially would take ˜278 hours of time;

In a theoretical worked example, wherein the model was trained fourtimes and algorithm training was linear:

-   0-25,000—˜69 hours—train;-   25,001-50,000 (25% completed by algorithm) 52 hours—retrain;-   50,001-75,000 (50% completed by algorithm) 35 hours—retrain;-   75,001-100,000 (75% completed by algorithm) 17 hours;-   100,000 images segmented using hybrid of algorithm and skilled    personnel—173 hours

The above simplified examples indicates that the segmentation algorithmwill be able to achieve the desired level of automation much faster withthe aid of retraining. In addition, this may be taken one step furtherby retraining the algorithm after each dataset is added to the trainingset. This could be achieved by using cloud infrastructures and eventdriven serverless computing platforms, such as AWS Lambdas. Showing theuser an updated set of labels after each retaining may dramaticallyreduce the time to create large amounts of data.

Moreover, most medical image segmentation applications require a veryhigh level of accuracy, and thus, the medical images may be used intheir original full resolution. However, in cases where there is aninherent need to look at the whole, or most of the, 3D scan in order todetect a pathology, e.g., an aneurysm, most 2D based approaches wouldnot be sufficient. Further, due to limitations in current hardware orprohibitive costs a 3D approach may not be applied to the fullresolution scans.

Thus, the methods described herein may down-sample the review volume tofind key features by using a D-Unet based architecture to segment thevasculature in CT scans, e.g., neuro CT scans. This architecture looksat small stacks of 2D images, e.g., 4 slices below and 4 slices above,thereby providing some small 3D contextual information. In the case ofaneurysm detection, the current approach may not be sufficient todistinguish between aneurysm and healthy vessels as it looks at only afew 2D images at a time, which may not be enough to achieve the contextneeded to be able to correctly identify aneurysms. This is mainlybecause the texture and general appearance of aneurysms isindistinguishable from other vasculature when looked at in isolation,e.g. in a few 2D images.

Being able to automatically identify and potentially locate and measureaneurysms, clots, and occlusions may revolutionize neurosurgery and savelives. For example, the methods described herein may use more advancedmethods that can look at the whole scan from a 3D perspective in orderto differentiate these abnormalities from the rest of the vasculature.Accordingly, the methods described herein may implement a two-stepapproach where the first step identifies the vasculature in the stack ofimages using a full resolution approach, and then a separate model wouldlook at a low resolution version of the scan in three dimensions in thesecond step. After obtaining the region where the aneurysm is in the lowresolution volume, the region may be co-registered with the highresolution version, such that the aneurysm may be segmented from thegeneral vasculature segmentation. This approach has a lot of potentialfor other high resolution 3D volume applications where there is a needto distinguish between similarly textured elements which require a muchlarger context in order to be correctly identified.

The preparation of images for the purposes of generating a model(physical or virtual) using real life medical images requires a certainamount of pre-filtering and improvement in order to generate an accuratemodel. Thus, a number of transformations must be performed to the imagesin order to dramatically improve the ultimate model quality.

For example, interpolation of images may be very amenable as a largedataset of existing images may be used to train the algorithm. This typeof problem is particularly suited to adversarial networks. Moreover,registration of images may be important as the number of cases thatinvolve multiple scanning modalities is increasing, and this there maybe a need to register CT→MRI images. For example, images from multiplescanning modalities may be registered by aligning two different datasetstogether, e.g., if a medical scan of a patient's head is provided and atumor is wanted from an MRI scan and a bone is wanted from a CT scan,landmarks may be picked that are visible on both MRI and CT scan inorder to register the pixels and voxels in the same position. Even MRIscans where the images have been taken in multiple perspectives/planesin a single session may require registration as the difference betweenthe planes may produce significantly different views of the patienthighlighting completely different aspects of the anatomy.

Focusing specifically on the integrations required to make theend-to-end possible rather than the individual processes themselves, thesystems and methods described herein focuses on how to integrate dataupstream and downstream of the platform.

This area may include all the integrations downstream such as theElectronic Medical/Health Records. Moreover, information from the EMR(potentially to associate with outcomes later c.f. Prognosis Section)may be collated, which would also include any upstream integrations suchas with couriers or printing bureaus. Key to the value in this area isthe idea of provenance of the data and showing the digital thread of theproduction of the model from data ingress right through to themanufactured object/virtual object and beyond.

While various illustrative embodiments of the invention are describedabove, it will be apparent to one skilled in the art that variouschanges and modifications may be made therein without departing from theinvention. The appended claims are intended to cover all such changesand modifications that fall within the true scope of the invention.

What is claimed:
 1. A method for multi-schema analysis of patientspecific anatomical features from medical images, the method comprising:receiving, by a server, medical images of a patient and metadataassociated with the medical images indicative of a selected pathology;automatically processing, by the server, the medical images using asegmentation algorithm to label pixels of the medical images and togenerate scores indicative of a likelihood that the pixels were labeledcorrectly; using, by the server, an anatomical feature identificationalgorithm to probabilistically match associated groups of the labeledpixels against an anatomical knowledge dataset to classify one or morepatient specific anatomical features within the medical images;generating, by the server, a 3D surface mesh model defining a surface ofthe one or more classified patient specific anatomical features;extracting, by the server, an isolated 3D surface mesh model of apatient specific anatomical feature comprising the selected pathologyfrom the 3D surface mesh model based on the metadata; and generating, bythe server, physiological information associated with the selectedpathology for the isolated 3D surface mesh model.
 2. The method of claim1, wherein the isolated 3D surface mesh model comprises a 3D surfacemesh model of an anatomical feature isolated from the one or moreclassified patient specific anatomical features based on the selectedpathology.
 3. The method of claim 1, wherein generating, by the server,physiological information associated with the selected pathology for theisolated 3D surface mesh model comprises: determining start and endpoints of the isolated anatomical feature; taking slices at predefinedintervals along an axis from the start point to the end point;calculating a cross-sectional area of each slice defined by a perimeterof the isolated anatomical feature; extrapolating a 3D volume betweenadjacent slices based on the respective cross-sectional areas; andcalculating an overall 3D volume of the isolated anatomical featurebased on the extrapolated 3D volumes between adjacent slices.
 4. Themethod of claim 1, wherein generating, by the server, physiologicalinformation associated with the selected pathology for the isolated 3Dsurface mesh model comprises: determining start and end points of theisolated anatomical feature and a direction of travel from the startpoint to the end point; raycasting at predefined intervals along an axisin at least three directions perpendicular to the direction of traveland determining distances between intersections of each ray cast and the3D surface mesh model; calculating a center point at each interval bytriangulating the distances between intersections of each ray cast andthe 3D surface mesh model; adjusting the direction of travel at eachinterval based on a directional vector between adjacent calculatedcenter points, such that raycasting at the predefined intervals occur inat least three directions perpendicular to the adjusted direction oftravel at each interval; and calculating a centerline of the isolatedanatomical feature based on the calculated center points from the startpoint to the end point.
 5. The method of claim 1, wherein generating, bythe server, physiological information associated with the selectedpathology for the isolated 3D surface mesh model comprises: calculatinga centerline of the isolated anatomical feature; determining start andend points of the isolated anatomical feature and a directional vectorfrom the start point to the end point; establishing cutting planes atpredefined intervals along the centerline based on the directionalvector from the start point to the end point, each cutting planeperpendicular to a direction of travel of the centerline at eachinterval; raycasting in the cutting plane at each interval to determinea position of intersection on the 3D surface mesh model from thecenterline; and calculating a length across the 3D surface mesh modelbased on the determined positions of intersection at each interval. 6.The method of claim 1, wherein generating, by the server, physiologicalinformation associated with the selected pathology for the isolated 3Dsurface mesh model comprises: determining start and end points of theisolated anatomical feature; taking slices at predefined intervals alongan axis from the start point to the end point; calculating across-sectional area of each slice defined by a perimeter of theisolated anatomical feature; and generating a heat map of the isolatedanatomical feature based on the cross-sectional area of each slice. 7.The method of claim 1, wherein generating, by the server, physiologicalinformation associated with the selected pathology for the isolated 3Dsurface mesh model comprises: determining start and end points of theisolated anatomical feature; calculating a centerline of the isolatedanatomical feature; determining a directional travel vector betweenadjacent points along the centerline; calculating a magnitude of changeof directional travel vectors between adjacent points along thecenterline; and generating a heat map of the isolated anatomical featurebased on the magnitude of change of directional travel vectors betweenadjacent points along the centerline.
 8. The method of claim 1, whereinthe generated physiological information associated with the selectedpathology for the isolated 3D surface mesh model comprises an associatedtimestamp, the method further comprising: recording, by the server, thegenerated physiological information and the associated timestamp; andcalculating, by the server, changes between the recorded physiologicalinformation over time based on associated timestamps, indicative ofprogression of the selected pathology.
 9. The method of claim 8, furthercomprising: calculating, by the server, a magnitude of the changesbetween the recorded physiological information over time; andgenerating, by the server, a heat map of the isolated anatomical featurebased on the magnitude of the changes between the recorded physiologicalinformation over time.
 10. The method of claim 1, wherein extracting, bythe server, the isolated 3D surface mesh model of the patient specificanatomical feature comprising the selected pathology from the 3D surfacemesh model based on the metadata comprises: isolating an anatomicalfeature from the one or more classified patient specific anatomicalfeatures based on the selected pathology; analyzing features of theisolated anatomical feature with an anatomical feature database toidentify one or more landmarks of the isolated anatomical feature;associating the one or more identified landmarks with the pixels of themedical images; and generating a 3D surface mesh model defining asurface of the isolated anatomical feature comprising the identifiedlandmarks.
 11. The method of claim 10, further comprising identifying,by the server, a guided trajectory for performing a surgical procedurefrom a surgical implement database based on the selected pathology andthe one or more identified landmarks; and displaying the guidedtrajectory to a user.
 12. The method of claim 1, further comprising:receiving, by the server, patient demographic data; identifying, by theserver, one or more medical devices from a medical device database basedon the patient demographic data and the generated physiologicalinformation associated with the selected pathology for the isolated 3Dsurface mesh model; and displaying the identified one or more medicaldevices to a user.
 13. The method of claim 1, further comprising:receiving, by the server, patient demographic data; identifying, by theserver, one or more treatment options from a surgical implement databasebased on the patient demographic data and the generated physiologicalinformation associated with the selected pathology for the isolated 3Dsurface mesh model; and displaying the identified one or more treatmentoptions to a user.
 14. The method of claim 1, wherein extracting, by theserver, the isolated 3D surface mesh model of the patient specificanatomical feature comprising the selected pathology from the 3D surfacemesh model based on the metadata comprises: isolating an anatomicalfeature from the one or more classified patient specific anatomicalfeatures based on the selected pathology; analyzing features of theisolated anatomical feature with an anatomical feature database toidentify one or more landmarks of the isolated anatomical feature;analyzing features of the one or more landmarks with a referencefracture database to detect a fracture of the isolated anatomicalfeature; and generating a 3D surface mesh model of the isolatedanatomical feature comprising the one or more identified landmarks andthe detected fracture.
 15. The method of claim 14, further comprisingmatching the 3D surface mesh model of the isolated anatomical featureagainst the reference fracture database to classify the detectedfracture.
 16. The method of claim 1, further comprising: delineating, bythe server, the classified one or more patient specific anatomicalfeatures into binary labels; separating, by the server, the binarylabels into separate anatomical features; and mapping, by the server,the separate anatomical features to original grey scale values of themedical images and removing background within the medical images, andwherein the generated 3D surface mesh model defines a surface of theseparate anatomical features, or comprises a volumetric render definedby mapping specific colors or transparency values to the classified oneor more patient specific anatomical features.
 17. The method of claim 1,wherein the segmentation algorithm comprises at least one of athreshold-based, decision tree, chained decision forest, or neuralnetwork method.
 18. The method of claim 1, wherein the physiologicalinformation associated with the selected pathology comprises at leastone of diameter, volume, density, thickness, surface area, HounsfieldUnit standard deviation, or average.
 19. A system for multi-schemaanalysis of patient specific anatomical features from medical images,the system comprising a server and configured to: receive medical imagesof a patient and metadata associated with the medical images indicativeof a selected pathology; automatically process the medical images usinga segmentation algorithm to label pixels of the medical images and togenerate scores indicative of a likelihood that the pixels were labeledcorrectly; use an anatomical feature identification algorithm toprobabilistically match associated groups of the labeled pixels againstan anatomical knowledge dataset to classify one or more patient specificanatomical features within the medical images; generate a 3D surfacemesh model defining a surface of the one or more classified patientspecific anatomical features; extract an isolated 3D surface mesh modelof a patient specific anatomical feature comprising the selectedpathology from the 3D surface mesh model based on the metadata; andgenerate physiological information associated with the selectedpathology for the isolated 3D surface mesh model.
 20. The system ofclaim 19, wherein the isolated 3D surface mesh model comprises a 3Dsurface mesh model of an anatomical feature isolated from the one ormore classified patient specific anatomical features based on theselected pathology.
 21. A non-transitory computer-readable memory mediumconfigured to store instructions thereon that when loaded by at leastone processor cause the at least one processor to: receive medicalimages of a patient and metadata associated with the medical imagesindicative of a selected pathology; automatically process the medicalimages using a segmentation algorithm to label pixels of the medicalimages and to generate scores indicative of a likelihood that the pixelswere labeled correctly; use an anatomical feature identificationalgorithm to probabilistically match associated groups of the labeledpixels against an anatomical knowledge dataset to classify one or morepatient specific anatomical features within the medical images; generatea 3D surface mesh model defining a surface of the one or more classifiedpatient specific anatomical features; extract an isolated 3D surfacemesh model of a patient specific anatomical feature comprising theselected pathology from the 3D surface mesh model based on the selectedpathology; and generate physiological information associated with theselected pathology for the isolated 3D surface mesh model.